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MULTIFLUOR-FLUORESCENCE IN-SITU HYBRIDIZATION 
(M-FISH) IMAGING TECHNIQUES USING MULTIPLE 
MULTIBAND FILTERS WITH IMAGE REGISTRATION 

CROSS -REyZRENCE TO RELATED APPLICATION 

This application claims priority from the following U.S. 
provisional application, the disclosure of which is incorporated by- 
reference in its entirety for all purposes : 

Application No. 60/029,314, filed October 25, 1996, of Jacques 
Ginestet, entitled "MULTI FLUOR -FLUORESCENCE IN-SITU HYBRIDIZATION (M-FISH) 
IMAGING TECHNIQUES USING MULTIPLE MULTIBAND FILTERS WITH IMAGE 
REGISTRATION. " 



BACKGROUND OF TEE INVENTION 
The present invention relates generally to fluorescence 
microscopy, and more specifically to improvements in image alignment where 
multiple multiband filters are used in M-FISH techniques. 

M-FISH (which stands for raultif luor-FISH, multiplex- FISH, 
5 mnlticolor-FISH, or multispectral-FISH) is a technique by which a number 
of fluorochromes (sometimes referred to as fluorescent dyes, or sitr^sly 
dyes) are used in what is otherwise a standard FISH (Fluorescence In-Situ 
Hybridization) procedure. A FISH san^le is prepared by using multiple 
probes, each of which binds to a different DNA sequence in the chromosomes 
10 in the san^le. Each probe is labeled with a different dye or combination 
of two or more dyes . 

A given dye is characterized by an excitation (absorption) 
spectrum and an emission spectrum. The excitation and emission spectra 
are also sometimes referred to as the excitation and emission bands. 
15 Accordingly when the dye is irradiated with light at a wavelength within 
the excitation band, the dye fluoresces, emitting light at wavelengths in 
the emission band- Thus when the sample is irradiated with excitation 
radiation in a frequency band that excites a given dye, portions of the 
sample to which the probe labeled with the given dye is attached 
20 fluoresce. If the light emanating from the sample is filtered to reject 
light outside the given dye*s emission band, and then imaged, the image 
nominally shows only those portions of the sample that bind the probe 
labeled with the given dye. 

FIG. 1 is a schematic showing a representative epi- illuminated 
25 fluorescence microscope system 5 for generating a single FISH image of a 
sample 10. The optical arrangement in system 5 includes an excitation 
filter 12 (shown as one of several such filters on a filter wheel) , a 
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dichroic mirror 15, a microscope objective 17 (say 60-lOOX) , and an 
emission filter 20 (sometimes also referred to as a barrier filter) . 
Excitation radiation from a source 25 passes through excitation filter 12, 
is largely reflected by dichroic mirror 15 and proceeds through the 
5 microscope objective 17 to sample 10. The excitation light traveling 
toward the sample is shown schematically by hollow arrowheads. 
Fluorescent radiation emitted from sample 10 passes back through 
objective 17, through dichroic mirror 15, through emission filter 20 to 
form an image in an image plane 30. The fluorescent light traveling away 
10 from the sample is shown schematically by solid black arrowheads. The 

image is digitized by a CCD camera 3 2 and the digitized image is sent to a 
computer 35 for subsequent processing. 

If the filters have single passbands, the particular filters 
and dichroic mirror are specific to a single dye in the sample. Images 
15 for other dyes in the sample are acquired by substituting optical elements 
configured for the excitation and emission bands for each other dye. The 
dichroic mirror and the emission filter are typically rigidly mounted to a 
supporting structure 40 (shown in phantom) , often referred to as a cube, 
with multiple cubes being movable into and out of the optical path. 
20 Oppositely directed arrows 42 represent a suitable mechanism such as a 

rotatzLble turret or a detent ed slide mechanism. The multiple excitation 
filters are typically deployed on a rotatable filter wheel (as shown) . 

The fact that the different images are produced by moving 
different cubes into the image path inevitably causes lateral and focus 
25 shifts and the like, thereby leading to misregistration of the images. 

While there are well-known image registration techniques, such techniques 
tend not to be robust. Since the different images represent fluorescence 
from different dyes, whose probes are attached to different portions of 
objects in the sample, attempts to register the images are prone to 
30 failure under at least some circumstances. • 

A number of variations of the instrumentation are well 
established in the prior art. For example, [Castleman93} discloses the 
use of a color CCD camera, multiband excitation and emission filters, and 
a polychroic mirror for simultaneously digitizing emissions from specimens 
35 laibeled with three fluorescent dyes. The excitation filter passes three 
narrow bands corresponding to the excitation bands of the three dyes, the 
polychroic mirror reflects the three excitation bands while transmitting 
the corresponding emission bands, and the emission filter only passes 
wavelengths falling within the three emission bands. 
40 As described in (Castleman93 ] , there is significant cross -talk 

betWMn the fluorescence channels due to the inevitable overlap among 
dyes' emission spectra and the camera's sensitivity spectra. This is 
addressed by an image processing step, referred to as color compensation, 
based on a predetermined knowledge of how each of a given dye's emission 
45 is recorded in each of the camera's RGB channels. A 3x3 matrix is 

determined, and the inverse of this matrix is applied to measured RGB 
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values to eliminate the effect of color spread among the camera RGB 
channels . 

Using a color camera and the multiband filters provides three 
images with the same cube, and thus avoids the registration problem. On 
the other hand, color cameras normally provide lower resolution and lower 
sensitivity than monochrome cameras. [Bornf leth96] compares the 
traditional technique of acquiring three monochrome images and the 
technique using a color camera and multiband filters, and concludes that 
the results are comparable. 

The use of a single cube with a color camera is limited to 
three dyes, however, whereas it is often desired to have more than three 
dyes. For example, five to seven dyes could provide a significantly 
larger number of possible combinations. While it may be possible to 
design a single cube configured for this many dyes, this would create 
15 additional challenges to recognize uniquely which dye or combination of 
dyes contributes to each camera channel. The already stringent optical 
demands would become even more stringent. It is necessary to have 
narrowband emission filters since the polychroic mirror typically 
transmits a small percentage (say 5% to 10%) of the much stronger 
20 excitation radiation that is inevitably reflected and scattered from the 
microscope surfaces and the sample. 

While it would be possible to extend the multiband approach of 
(Castleman93] to more than three dyes by providing two or more multiband 
cubes and multiband excitation filters, each configured for three dyes, 
25 the problem of image registration would again arise. 

SUMMARY OF THE INVENTION 
The present invention provides techniques for acquiring and 
analyzing M-FISH images in a manner that allows for proper image 
registration and yet is easily scaleable for large numbers of dyes. The 
invention has applicability with epi-illumination and trans-illumination 
optical systems. The invention further provides techniques for combining 
a plurality of registered images to provide labeled images. 

In short, with respect to epi-illumination embodiments, an 
aspect of the invention recognizes that it is impractical to configure a 
35 multiband filter cube for more than four or five dyes, and therefore takes 
the approach of having multiple filter cubes for different siibsets of the 
entire set of desired dyes. The invention addresses the registration 
problem by configuring the different multiband cubes so that pairs of 
cubes have a common dye for which they are configured. Accordingly, a 
first set of images generated with a first cube includes an image that 
corresponds to (contains the same features as) one of the images in a 
second set of images generated with a second cube. This allows the first 
set of images to be precisely and accurately aligned with the second set 
of images . 



30 



40 
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In a specific embodiment, there are two cubes, each configured 
for four dyes with one of the dyes being the same for the two ciibes. 
Further, it is preferred to have the dye that is common to the two cubes 
to be a counterstain that stains all DNA and therefore provides an image 
5 with substantially all the objects, thereby allowing the most precise 

registration. The present invention further contemplates performing color 
compensation in order to allow accurate quantitative analysis of the 
images . 

This approach allows considerable design flexibility. For 
10 example, by selecting a dye that has broad excitation and emission spectra 
as the common dye between the two cubes, the corresponding bands on the 
cubes do not have to be as well optimized, thereby allowing the other 
bands on the cubes to be more fully optimized. 

The invention also has applicability for trans -illumination 
15 embodiments where multiple multiband emission filters are interposed in 

the path between the sample and the image plane. The invention addresses 
the registration problem by configuring the different multiband emission 
f iXters so that pairs of filters have a common dye for which they are 
configured. Accordingly, a first set of images generated with a first 
20 multiband emission filter includes an image that corresponds to (contains 
the same features as) one of the images in a second set of images 
generated with a second multiband emission filter. This allows the first 
set of images to be precisely and accurately aligned with the second set 
of images . 

25 According to further aspects of the invention, a labeled 

image, i.e., a map of the various dye combinations, is generated from a 
set of registered color compensated dye images . This could apply to 
images registered as discussed above, images otherwise registered, or 
images generated with a single optical setup in the image path, i.e., 

30 without interposing multiple interchangeable elements in the image path. 

A method for generating a labeled image typically includes a 
normalization of the pixel intensities within each image, followed by a 
transformation to ratio images. The latter entails dividing each pixel 
value in a given normalized color compensated dye image by the sum, taken 
35 over the images for all the different dyes, of the pixel values for that 
pixel position. The ratio images may then be thresholded, and any pixel 
which is non-zero for a given ratio image can then be considered as having 
been labeled with the corresponding dye. Each pixel can then be colorized 
according to the combinations of dyes considered to have labeled that 
40 pixel. 

A further understanding of the nature and advantages of the 
present invention may be realized by reference to the remaining portions 
of the specification and drawings. 
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BRIEF DESCRIPTION OF THE DRAWINGS 
FIG. 1 is a schemanic showing a representative epi illuminated 
fluorescence microscope system within which the invention may be embodied; 

FIG. 2 is a high-level flowchart of a method according to the 
5 present invention; 

FIGS. 3A-3G show the excitation and emission spectra for seven 
representative fluorescent dyes; 

FIG- 3H shows a composite view of the spectra; 

FIGS. 4A-4C show relative spectral outputs of tungsten, xenon, 
10 and mercury lamps; 

FIG. 5A shows the bandpass spectra for the polychroic mirror 
and the emission filter in a first one of the cubes according to the 
invention; 

FIG, SB shows the bandpass spectra for the polychroic mirror 
15 and the emission filter in a second one of the cubes according to the 
invention; 

FIG. 6 is a flowchart showing additional details of the image 
acquisition step shown in FIG, 2; 

FIG. 7 is a flowchart showing additional details of the image 
20 registration and image combination steps shown in FIG- 2; 

FIG. 8 is a flowchart showing additional details of the image 
combination step shown in FIG. 2; 

FIG. 9 is a schematic of a representative trans -illumination 
fluorescence microscope system within which the invention may be embodied; 
25 and 

FIG. 10 is a schematic of a representative darkfield 
fluorescence microscope system within which the invention may be embodied. 

DESCRIPTION OF SPECIFIC EMBODIMENTS 

1. Overview 

3P "G- 2 is a high-level flowchart of a method of generating a 

set of fluorescence images according to the present invention. The basic 
steps are shown in a particular order, but some of the steps may be 
performed in a different order, or portions of different steps may be 
interleaved. 

35 The method is performed with apparatus, generally as shown in 

FIG. 1, except using polychroic mirrors and multiband emission filters 
configured in accordance with the invention in order to permit desired 
large number of fluorescent dyes to be used while still permitting 
accurate image registration among the individual images. The description 

40 will be primarily with reference to epi -illumination embodiments which use 
a wavelength- selective mirror to separate the excitation radiation from 
the fluorescent emissions from the dyes. 

A particular implementation uses an Olympus BX60 microscope, 
which includes a built-in universal reflected light fluorescence vertical 

45 illuminator. Other Olympus models (BX40 and BX50) must be retrofitted 
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with such an 'illuminator as an add-on accessory. The illuminator, which 
is located above the microscope objective turret, is configured with an 
industry -standard dovetail mount to accommodate viewing optics and a 
camera, and accommodates a cube turret. The microscope and its 
5 accessories are available from the Precision Instrument Division of 

Olympus America, Inc, located in Melville, New York. The particular 
implementation uses a cooled monochrome slow- scan CCD digital camera, 
marketed under the mark SENSYS, and available from Photometries Ltd, 
located in Tucson, Arizona. The particular implementation also uses an 
10 excitation filter wheel. 

In short, the invention uses a plurality of N multiband cubes, 
each comprising a polychroic mirror and a multiband emission filter. Each 
cube is configured for an associated one of N siibsets of the total set of 
dyes, with the i— subset having a plurality of distinct dyes (i ranges 
15 from 1 to N) . The subsets are chosen so that the subset and the 

(j+l)th subset have one dye in common, referred to as the common dye 

(j ranges from 1 to N-1) . In a specific embodiment, a given polychroic 
mirror has passbands matched to the emission spectra of the dyes in the 
associated subset, while reflecting light at the wavelengths used for 
20 excitation. The associated multibcoid emission filter also has passbands 
matched to the emission spectra of the dyes in the associated subset. 

The excitation filters are matched to the reflection bands of 
the polychroic mirrors, which are matched to excitation spectra of the set 
of dyes. Since the excitation filters are not disposed in the path from 
25 the scunple to the image plane, separate single-band filters can be used 
without raising image registration problems. 

In possible alternative embodiments, the polychroic mirror 
would be configured to transmit light at the wavelengths used for 
excitation, and to reflect light at the wavelengths at which the dyes 
30 emit. In such embodiments, the associated multiband emission filter would 
still have passbands matched to the emission spectra of the dyes in the 
associated subset, but this would correspond to the reflection bands of 
the associated polychroic mirror. Similarly, the excitation filters would 
still be matched to excitation spectra of the set of dyes, but this would 
35 correspond to the transmission bands of the polychroic mirrors. 

A calibration procedure 50 is performed to develop a set of 
parameters that can be used in a later processing step (color 
compensation) used to account for the fact that emissions from one dye 
show up in an image that is nominally intended to contain only objects 
40 labeled with a different dye. Put another way, in an ideal world, when 

the excitation filter for a given dye is in place, the resultant image 
would only show portions of the sample that bind the probe labeled with 
the given dye. However, as described above, there is cross-talk, and it 
is preferred to process the images to correct for such cross -talk. This 
45 is discussed in detail below . in a section entitled "Color Compensation." 
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15 



20 



This is followed by an image acquisition phase 55 where images 
nominally corresponding to the different dyes are produced. An image for 
a given dye is generated by using the cube associated with that dye's 
subset and the appropriate excitation filter. It is noted that multiple 
images will be acquired for each dye that is a common dye. Corrections 
such as a darkfield correction may be applied to the images prior to 
further processing of the images. 

The sets of images taken with the different cubes are then 
subjected to an image registration step 60 that makes use of the images 
for the common dyes. The images, so registered, are then subjected to a 
color compensation step 65, which makes use of the color compensation 
calibration information to correct pixel values in each image so that they 
represent, as well as is practical, the presence or absence of emission 
from that image's corresponding dye. The color compensated images are 
sometimes referred to as pure dye images. The registered and color 
compensated images are then preferably combined at a step 70 that provides 
a map of the various dye combinations. 

The apparatus typically operates under computer control to 
ensure proper energization of the light source, stepping the cube turret 
and the excitation filter wheel, and operating the camera. The computer 
that controls the apparatus may be the same computer that processes the 
images . 



2. Dye Characteristica 

From a theoretical point of view, the choice of a set of dyes 
25 requires only that the dyes have distinguishable excitation and emission 
spectra. A possible criterion is that at any given pair of dyes have to 
have excitation peaks or emission peaks that differ by more than some 
threshold amount (say 20 nm) , defined by the filters* capabilities to 
differentiate . 

30 From a practical point of view, the dyes are more or less 

imposed by molecular biology constraints. For example, in [Speicher96] 
the dyes of choice are DAPI, FITC, Cy3 , Cy3 . 5 , Cy5 , Cy5 . 5 , and Cy7 . 
Approximate values of their respective excitation and emission maxima are 
summarized in the table below; 



Dye ( fluorochrome) 


Excitation Peak 


Emission Peak 


DAPI 


350 


450 


FITC 


490 


520 


Cy3 


550 


570 


Cy3 .5 


580 


595 


Cy5 


650 


670 


CyS.S 


680 


700 


Cy7 


755 


780 



35 



40 



SUBSTITUTE SHEET (RULE 26^ 
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FIGS. 3A-3G show the excitation and emission spectra for these 
representative fluorescent dyes. FIG, 3H shows the spectra jxixtaposed on 
a single plot. The emission spectrum of a dye is substantially 
independent of the excitation wavelength used to excite the dye, but for 
5 efficiency's sake, it is preferred to excite the dye with a range of 
wavelengths near the maximum in the excitation spectrum. 

These dyes have to be treated pretty much as a given of the M- 
FISH problem. However, the wet preparation (namely the concentrations of 
the various dyes) still is preferaibly adjusted so that the various dyes, 

10 as observed through the chosen filters, have reasonable brightnesses 
(i.e., require reasonable exposure times). Furthermore, the relative 
total brightness of each dye, as observed with the selected sets of 
filters, is preferably roughly equal to that of the other dyes (say within 
a factor of 2 or 3) so as to avoid one signal overpowering the others 

15 (issues of dynamic range, non-linearity, saturation, etc) , 

FIGS. 4A-4C show relative spectral outputs of tungsten, xenon, 
and mercury lamps. The choice of excitation light source is practically 
limited to mercury or xenon arc lamps; mercury is brighter, while xenon 
has a more unifonn spectrum. A priori, xenon would be preferable because 

20 of the large numbers of dyes to be used. A 150W lamp is preferred to 

compensate for the lower output efficiency, although a 75W xenon lamp may 
be sufficient. The lamp is preferably provided with a filter to block IR 
above SOOnm. 

3 . C\ibe Characteristics 

25 As can be seen from the spectra in FIGS. 3A-3G, most of the 

dyes (DAPI is an exception) have their excitation peaks close to their 
emission peaks. Further, the emission peak of one dye tends to be close 
to the excitation peak of another. Accordingly, the design of a single 
ctibe that passed the emissions from the seven dyes while 

30 blocking/ref lecting the excitation light would be extremely difficult 

(requiring very narrow passbands) . On the other hand, if one has to deal 
with only a subset of the dyes at a time, the design constraints are 
considerably relaxed. 

FIG. 5A shows the bandpass spectra for the first cube*s 

35 polychroic mirror (heavier line with troughs that exhibit non-zero 

trsmsmission between transmission peaks) and the emission filter (lighter 
line, with substantially zero transmission between peaks) ; FIG. SB shows 
the second cube's spectra. The first cube is configured for the subset of 
dyes consisting of DAPI, FITC, Cy3.5, and Cy5 . 5 , while the second cube is 

40 configured for the subset of dyes consisting of DAPI, Cy3 , Cy5 , and Cy7 . 

DAPI is the common dye. One point to note in the specific 
design is that the emission filters for DAPI are at different wavelengths 
in the two cubes (450-470 nm in the first cube and 475-490 nm in the 
second) . This exploits DAPI * s broad emission band, allowing flexibility 
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in the placement of the DAPI band, thereby allowing better placement of 
the bands for the other dyes in each of the subsets , 

The excitation filters are preferably implemented as a set of 
single-band filters (e.g., interference filters), which allows a given 
5 filter to pass excitation light in a very precise and narrow band (say 

10-20 nm) . The excitation filters should be matched to the set of rubes 
in the sense that the excitation light passed by a filter should not be 
within the passband of a cube with which it is to be used. Since the 
passband of a cxibe is ultimately defined by the emission filter passbands 

10 for that cube, this means that the excitation filters' passbands cannot 
overlap the emission filters • passbands , 

In embodiments where the polychroic mirrors pass the 
excitation light and reflect the fluorescent emissions, it is still true 
that the excitation filters • passbands cannot overlap the emission 

15 filters* passbands. However, in such cases, the emission filters* 

passbands will fall outside the polychroic filters ' passbands and the 
excitation filters' passbands will fall within the polychroic mirror's 
passbands. 

It is in principle possible to use the same excitation filter 
20 with more than one cube if two dyes associated with different cubes have 
sufficiently close excitation spectra. The dyes should have sufficiently 
different emission spectra that each dye*s emissions will be passed only 
by that dye ' s associated cube . 

Note that the polychroic mirrors do not reflect 100% of the 
25 excitation light, as evidenced by the fact that their transmission never 
falls to zero in the regions between the passbands . This makes it 
important to have the associated multiband emission filter, whose 
transmission falls substantially to zero (say less than 10"*) between the 
passbands. 

3 0 There is a tradeoff between the number N of different cubes 

and the number of dyes for which each cube is configured. In the specific 
embodiment, where there are seven dyes, there are two cxibes, each 
configured for four dyes (including the common dye) . It is feasible to 
design a cube for four dyes; it is much more difficult to design a cube 

35 for seven dyes . While it would be even easier to optimize a cube for two 
dyes, that would lead to an excessive number of cubes. 

When there are more than two ciobes, the issue arises whether 
the same common dye should be used for more than two adjacent pairs. This 
is typically a design choice. Repeating a dye more than once could have 

40 the effect of necessitating more cubes, but there may be reasons to prefer 
a certain dye as the common dye between more than one adjacent pair. In 
the specific embodiment, the common dye is DAPI, which is a coiinterstain 
that stains all DNA. This means that the DAPI image in each set of images 
contains the most objects, and makes the registration even more robust. 

45 Further, DAPI has broader excitation and emission spectra, which relaxes 
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the design constraint on the cubes, making it easier to optimize the cube 
for the other dyes in each subset . 

4 . Image Acquisition and Correction 

PIG. 6 is a flowchart showing additional details of image 
5 acquisition step 55 (shown in FIG- 2) . The image acquisition commences 

(step 72) , and, as noted above, entails sequencing through the cubes 
(outer loop 75, bounded by step 75a and branch 75b) and, for each cube, 
sequencing through the excitation filters for the cxibe ' s associated dye 
subset (inner loop 80 bounded by step 80a and branch 80b) . Thus, the 
10 image acquisition step preferably includes, for values of i from 1 to N, 

illuminating the sample through an appropriate one of the excitation 
filters and the i^ polychroic mirror, and directing light emanating from 
the sample to the camera along an optical path that includes the i— 
polychroic mirror and the i— emission filter. Thus each of the images in 
15 the plurality of M^^ images corresponds to a respective one of the 

distinct dyes. The acquired image is referred to as an illuminated image, 
and its acquisition is shown schematically as a step 82 . 

It is preferred to perform a darkfield correction on the 
acquired images to account for "hot" pixels in the camera and dark 
20 current. This entails acquiring images with no illumination (step 85), 
typically effected by turning the source off and having one of the 
positions on the excitation filter wheel being opaque so as to act as a 
shutter. These images, referred to as darkfield images, are subtracted 
from the illuminated images (step 90) . Each darkfield image is preferably 
25 characterized by the same acquisition parameters (exposure time and 
digitization parameters) as the illuminated image from which it is 
sxibtracted. If these parameters are the same for all illuminated images 
for a given cube, only one darkfield imaige need be acquired for the cube, 
and siibtracted from all the illuminated images for that cube. At the 
30 other extreme, where the acquisition parameters are different for the 

different images for a given cube, it is preferable to acquire multiple 
darkfield images. The possibility of acquiring only a single darkfield 
image for a set of illuminated images for a given cube is represented in 
the figure by the parenthetical notation "if needed." 
35 While the flowchart shows the darkfield corrections being 

performed after all the images are acquired, there may be advantages to 
interleaving the image acquisition and darkfield correction. For example, 
storage requirements could be reduced if the darkfield image were to be 
subtracted from its corresponding illuminated image as soon as the two 
40 images are available. Depending on the processing power of the computer, 

this may he possible without unduly prolonging the image acquisition step. 
On the other hand, if the image processing is to be offloaded to another 
computer, it may be preferred to shorten the image acquisition step by 
doing all the darkfield corrections after the images are acquired (as 
45 shown in the figure) . 



BNSDOCtO: <WO 9817992A2J_> 



wo 98/17992 



PCT/US97/19464 



11 

5 . Image Registration 

FIG. 7 is a flowchart showing additional details of image 
registration step 60. As discussed above, image registration between the 
Mj images taken with the ctibe and the Mj^^ images taken with the 

5 {j+i)th cxibe relies on the fact that each set includes an image taken with 

a filter set configured for the same (common) dye. This ensures that the 
two images contain the same objects, thereby making image correlation 
techniques very robust. It is convenient to number the different subsets 
in a linear array with each one of (N-1) adjacent pairs sharing a common 

10 dye. Thus the first and second subsets share a common dye, the second and 
third subsets share a common dye, etc. Equivalently , since the cubes are 
uniquely associated with the subsets of dyes, the cubes can be thought of 
as defining (N-1) adjacent pairs. 

The image registration begins by selecting the relevant 

15 subsets of images (preferably darkf ield corrected) , namely those having 

been taken with adjacent pairs of the cubes. The image registration thus 
entails sequencing through the subsets of images for adjacent pairs (loop 
100, bounded by step 100a and branch 100b) . For each adjacent pair, the 
images corresponding to the common dye are registered. In a specific 

20 embodiment, this entails finding the centers -of -gravity of the objects in 
the common dye images (step 102) , and determining a set of registration 
parameters that provide the best correlation between the sets of object 
centers (step 105) . This correlation may be a least squares fit. In a 
current embodiment, x and y translational offsets provide adequate 

25 registration; other embodiments could benefit from determining additional 
parameters representing rotation, magnification, and skew. 

Once the offset between the common dye images is determined, 
the offset values are applied to all the images in one of the subsets to 
register them to the images in the other subset (step 107) . Once all the 

30 image subsets have been registered, the images are color compensated to 
produce the images can be combined in any desired manner to determine 
which portions of the sample contain various ones of the set of 
fluorescent dyes. 

6 . Color Connoensation 

35 Color compensation has been outlined in general terms in 

(Castleman93] and [Castleman94 ] . (See also [Bornf leth96] for a somewhat 
different treatment) . Given the actual emission spectra of the selected 
dyes, and the actual sensitivity spectra of the selected filter sets (and, 
to a lesser degree, of the camera) , it is generally impossible to obtain 

40 complete isolation of each dye. Put another way, one cannot obtain images 
representative of only a single dye; spectral overlap spreads each dye's 
contribution across the various images . 

Consider, for the sake of simplicity, the case of three dyes ; 
the following discussion can easily be extended to any number of dyes. 

45 Assume that for such preparations, there are three different filter sets 
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(S^, S-y, S3), i.e., sees of matched excitation, dichroic and emission 
filters . 

If a sample contains only dye F^^, and is imaged with each of 
the three filter sets, there are three resulting images i^^l' ^21' ^31^ • 
5 For each pixel, it is possible to compute the intensity ratios between 
images (i.e. , 

l3_3_(i, 3) / i'^xz^^' **■ ^21*^' ^31^^' j)]). and then 

compute the average over all pixels above a certain threshold (to avoid 
signal-to-noise problems, divide-by- zero problems, etc) . This gives three 
10 factors (C^^^, ^22,' ^31^ representing the proportion of emissions from dye 

F^ which made its way through the various filters sets (note that 

I31 = 1) • Similarly for dyes F2 and F3 , one obtains {C^2' ^22* ^32^ 
and (03^3, C23, C33) respectively. 

It is possible to define a matrix C, the color spread matrix, 
15 where each element Cj^j represents the proportion of the total measured 
brightness of emissions from dye i which was observed by filter set j : 



20 



;ii 
:2i 

-31 



:i2 
-22 

-32 



i 

C33 I 



Note that the columns of C sum to 1; this is intuitively desirable, so 
that the discussion can be in terms of percentages, but is actually not 
required; the only thing required is that th.e coefficients be independent 
25 of absolute intensity and representative of relative intensities. 

Then, if one images a preparation containing all three dyes, 
for each pixel, given the true dye amounts present at that point 
^^l'^2'^3^' would measure the values (i3_,i2,i3) such that 



30 



I ^11 ^12 ^13 
I ^21 ^22 ^23 
I ^31 ^32 ^33 



or 



[Eq. 1] 



And therefore 



F = C 



(Eq. 2] 



That is, if, at each pixel, the observed vector I is multiplied by the 
35 inverse of the color spread matrix, C'^, called the color compensation 
matrix, the true dye brightness values are recovered. 

Obviously, this technique will work if C is well conditioned; 
that is if C can be inverted, and if the solution is stable with respect 
to small variations of the observed values or of the coefficients of C. 
40 Such a case can be tested by doing a singular value decomposition (SVD) 
of C (Press921 . Note that in the case of square matrices (which C is) , 
singular values and eigenvalues are essentially the same thing. 
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Briefly stated SVD is based on che fact that any MxN matrix 
whose number of rows M is greater or equal to its number of columns N can 
be written as the product of an MxN column- orthogonal matrix U, an NxN 
diagonal matrix W with positive or null elements (the singular values) , 
5 and the transpose of an NxN orthogonal matrix V. That is, 

C = U * W * [Eq. 3] 

where : 

W is diagonal, with each a O; 

uT*u = V*V^ = V^*V=rl (orthogonal conditions) ; and 
10 1 is the identity matrix. 

It follows that 

C"^ = V * W« * TJ^ [Eq. 4] 

where W» is diagonal, with each w'^ = 1/wj^ . 

Obviously, this is defined only if all the singular values are non-zero; 
15 in such case, C is called non- singular , and a unique solution can be 
obtained: 

PsV*W« * XJ^ * X [Eq. 5] 

If some of the singular values are very small, then C is said to be ill- 
conditioned, and even though an inverse exists, the solutions are unstable 

20 (i.e., a small change in the observed values I creates a large change in 

the calculated values F) . However, even in such a case, if C is singular 
(one or more of its singular values is zero) or ill-conditioned (one or 
more of its singular values is very small) , one can find the solution 
which minimizes the residual JC.F-I|; this is not "the" unique solution 

25 (which does not exist in such a case), but is often as good. It is 

obtained by using a modified matrix W" , where 1/w^ is replaced by zero if 
w^ is zero (or very small) . In this case, this solution, optimal in the 
least square sense, is given by: 

PssV*W"*XJ^*I [Eq. 6] 

30 Therefore, it is preferred to perform a SVD on C, to eliminate null or 
very small singular values, and to then solve [Eq. 6] for the solution 
which minimizes the residual ( [Press 92] , page 62) . 

In terms of computation times, note that this SVD needs to be 
computed only once; thereafter, only a direct matrix multiplication needs 

35 to be performed (albeit for each pixel) . As an order of magnitude, 

performing a NxN matrix multiply at each pixel, will require about as much 
computing power as performing an NxN convolution of the image (hence a few 
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seconds to tens of seconds for values of N around 7) , Algorithms for the 
SVD (and even source code) are readily available (e.g., [Press92l ) . 

Finally, it will be evident that this whole approach can 
easily be extended to any number N of dyes, simply by observing them with 
5 an equal number N of filter sets (thus giving a NxN color spread matrix) . 

Actually, one could use M filter sets, with M > N; this would give an 
over determined problem, but one that could potentially be much more stable 
than with M = N. Such an overspecif ied can be solved by the very same SVD 
technique [Press92l . 

10 The color spread matrix (and hence the color compensation 

matrix) depends on the dyes, the filter sets, the lamp used for 
excitation, and the camera. Recalibration is generally needed if any of 
them changes. This is because such changes are non- linear (i.e., do not 
affect all raw images identically) and are uncalibrated (i.e., there is no 

15 analytical expression of these changes) . Thus it is generally difficult 
to account for this in the set of linear equations. 

It should be noted that the determination of the color spread 
matrix will require the acquisition of images (where N is the total 
number of dyes used; five to seven in practice, thus 25 to 49 images to 

20 capture) . However, the total number of images to acquire for calibration 
can be reduced to N if a calibration scene containing easily identifiable 
single -labeled chromosomes can be obtained. Such a scene would contain 
some chromosomes labeled with only dye F^^, some labeled with only dye F2 , 
etc, each easily identifiable by means other than their color. 



25 7 . Tm»ey«> Combination 

In the case of M-FISH, one is normally interested in 
generating a labeled image, i.e., a map of the various dye combinations. 
There are a number of ways to approach this. In essence, what is desired 
is to determine, for each pixel position, which of the pure dye images has 
30 a sufficient contribution to signify the presence of that dye at the 
corresponding site in the sample. 

FIG. 8 is a flowchart showing additional details of image 
combination step 70 shown in FIG, 2. In the case of M-FISH, one is 
normally interested in generating a IcdDeled image, i.e., a map of the 
35 various dye combinations. There are a number of ways to approach this. 
In essence, what is desired is to determine, for each pixel position, 
which of the pure dye images has a sufficient contribution to signify the 
presence of that dye at the corresponding site in the sample. 

One approach is simply to threshold the color compensated pure 
40 dye images, and then, for each pixel position, determine which pure dye 
images have a non- zero value at that pixel position. This approach is 
simple, but it may be difficult to determine an appropriate threshold when 
the images differ in intensity. Differences in intensity can arise as a 
result of a number of factors, including the vagaries of wet preparation 
45 and possible non -uniformity of illumination. 
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Accordingly, it is generally preferred to perform some sort of 
normalization (step 110) before determining the relative values of the 
different images at a given pixel position. One approach to normalization 
is, for each pure dye image, to determine the maximum value in that image, 
and then normalize the pixels in that image to the maximum value in that 
image. This technique may be susceptible to spurious objects in the 
image, and so may be refined by normalizing each image to its average 
pixel value . 

A preferred technique for normalizing the pure dye images is 
to base the normalization on pixels that correspond to objects of interest 
in the images (i.e., chromosomes in the M-FISH case) . This is done by 
creating one or more masks (step 112) representing locations (i.e., having 
ON pixels) where there are expected to be objects. While finding the 
objects in the general case could entail significant image processing, it 
15 is facilitated in the special case where one of the dyes is DAPI, which 

stains all chromosomes. As described above, DAPI is preferably one of the 
dyes, and indeed the dye used for the common dye. In this case, either of 
the DAPI images can be thresholded and then used as a mask. 
Alternatively, the union of all the pure dye images (preferably 
20 individually thresholded) can be used as a mask. 

Since the images have been registered, non-zero pixels in each 
image that fall within the mask can be considered to belong to objects 
having that dye. Thus, the normalization of the images further includes, 
for each of the registered pure dye images, determining non-zero pixels 
25 falling within the mask (step 115) , and normalizing the values of only 

those non-zero pixels (step 117) . The average value of these pixels (or 
alternatively, the maximum value, the modal value, or even the total 
integrated intensity of all pixels belonging to objects) can then be used 
for normalization. 

The individual pure dye images, so normalized, are then • 
preferably transformed to ratio images (step 120); if a given pixel 
position has pixel values (f^^, f2'---fn^ the pure dye images, the 

pixel values in the pure dye images are transformed to (f^^/sum, 

f2/sum f jj/sum) , where sum=f3^+f2 + . . .f^. Thus each pixel value in a 

35 given normalized pure dye image is divided by the sum, taken over all the 
normalized pure dye images, of the pixel values for that pixel position. 

The ratio images are then thresholded, say at 25 or 30% (step 
125) , and any pixel which is non-zero for a given ratio image can then be 
considered as having been labeled with the corresponding dye. Note that 
40 this approach is much less sensitive to variations in illumination 

(shading effects) than the first approadtt of merely thresholding the pure 
dye images.. Each pixel is then colorized (step 130) according to the 
combinations of dyes considered to have labeled that pixel. 
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8. Tmacre Diaitization 

Clearly the pixel values acquired, both for the calibration 
and for the actual experiment, depend on the digitization parameters 
(gain, offset, exposure); if one were to use different such parameters, 
5 one would obtain images with very different pixel values, and hence, for 
example, very different color spread matrix, even though the same dye 
concentrations were present in the scene. These parameters are preferably 
taken into account in the method. 

First of all. note that the above formulation requires that 
10 the absence of dye gives zeros for the observed values (i.e., no dye means 
no signal) . Therefore, the first constraint is that the digitizer offset 
be set just about the clipping point (i.e., at a point where black pixels 
(i.e., background pixels) have a numerical value just above, just below, 
or equal to zero) . Secondly, for each image acquired, a darkfield 
15 correction is required (suiDtraction of a dark field image, e.g., shutter 

closed, otherwise acquired under the same conditions) . Finally, it may be 
necessary to also do a background subtraction (local, adaptive algorithm, 
e.g. , top-hat) . 

As far as exposure time is concerned, it is calibrated (and 
20 linear) by definition (i.e., if the pixel value is known for a given 

eaqjoaure time, it can be easily deduced for another exposure time, as long 
as no saturation occurs); however, with video cameras, changing the 
exposure time also introduces an offset shift. So, if the exposure time 
is changed to accommodate changes in signal levels between experiments, or 
25 between experiments and the calibration, an accompanying digitizer offset 
change is preferably made to make backgroiind pixels have a gray level of 
zero or close to zero. Hence another reason to set the offset just about 
the clipping point, to perform a darkfield correction, and to subtract the 
local background. 

30 To prevent loss of linearity in the system of linear 

equations, the digitization parameters are preferably set so that there is 
no saturation. This saturation is to be avoided both in the CCD (i.e., 
the e:q>osure is set too high; the well capacity is exceeded) or in the 
digitizer (i^e., the gain is set too high; the maximum pixel value is 
35 exceeded) . Therefore, the gain is preferably set at 1 (or very close 
to 1) ; larger values would risk saturation, lower values would prevent 
from detecting CCD saturation. Which specific numerical input • corresponds 
to unity gain will have to be calibrated, system by system (easily done 
with an oscilloscope, or even better a signal generator) . 

40 Another reason for keeping the gain fixed is that, for most 

digitizers, gain is not calibrated (i.e., it is not known to which gain a 
given numerical input corresponds) , is not very stable, and is not 
particularly repeatable. 

It should be noted here that the above formulation assumed 

45 that the exact same image acquisition parameters were used between 

calibration and actual data acquisition, and therefore that the exact same 
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image acquisicion paramecers were used to calibrate all dyes with a given 
filter set (e.g., dye F^^ captured with filter set S^^ used the same 
parameters as dye F2 with filter set S-j^, which are the parameters used for 
acquiring the raw data with S-^; however with S2 could use different 
5 parameters) . This is obviously too constraining in practice, and indeed 
does not have no be the case. 

It has already been established that the gain is preferably 
kept fixed at 1 and that the offset is preferably adjusted to operate just 
at the clipping point. This leaves the exposure time as the last 
10 remaining free parameter. As noted earlier, exposure time is calibrated 
(and linear) , so its impact can be accounted for. 

If e^j is the exposure time used to calibrate dye j with 
filter set i , the color spread matrix C can be brought back to the case of 
unity exposure by dividing each of its coefficients c^j by the 
15 corresponding exposure e^^ j . Therefore the result of the calibration is 
actually the modified color spread matrix (color spread matrix with 
unity exposure time) . 

If in the actual experiment, exposure e^^ is used to acquire sin 
image with filter set i, Eqs. 1 and 2 above become, respectively 

20 I = E ♦ Cj^ * F lEq. 7] 

F « C^'^ * B"^* I tEq, 8] 

where E is a diagonal matrix specifying the exposure time used to acquire 
each image (i.e., e^^j = 0 if i j, e^^ - exposure time used to capture 
image with filter set i) . 

25 Note that left multiplying by the matrix E is equivalent to 

multiplying each row of C^^ by the corresponding exposure, which is the 
same as multiplying the right hand side of each of the linear equations by 
the corresponding exposure. Therefore, in practice, one should left 
multiply the color spread matrix C^^ by the actual exposure times, and then 

30 proceed with the SVD. 

In conclusion, the digitizer gain is preferably kept fixed at 
or near 1, the offset is preferably adjusted to operate just at the 
clipping point, a darkfield correction is preferably applied to each 
image, local adaptive background sxibtraction preferably takes place, and 

35 the color spread matrix is preferably properly adjusted to accoxint for 
differences in exposure times . 

Note that another approach to the issue of the digitization 
parameters is to use a digital camera (instead of an analog camera) ; the 
gain there is easy to fix at 1 (as a matter of fact, that is all which is 

40 offered on. some cameras), there is no notion of offset, gray levels are 
linear with exposure, and darkfielding plus localized backgroiind 
subtraction will take care of the (small) pixel value shifts (dark current 
etc) which may occur with different exposure times. Also, by providing 10 
to 14 bits per pixel, this would improve the precision (and stability) of 
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the subsequent computations (while, in the end, the system might only keep 
8 bits of pure dye data, these would be 8 "good" bits) . The one 
disadvantage is that such digital cameras have relatively slow readout 
rates, hence would give slower frame rates than analog cameras. However, 
if the light levels were very weak, a choice of a digital, cooled CCD 
camera might be called for in any case. 



9 . Practical Considerationg 

In practice, one obviously tries to choose the dyes and the 
filter sets so that the matrix C is as close to the identity matrix as 

10 possible (and hence so is C'^) . As a matter of fact, with the currently 
standard way of acquiring FISH images (monochrome camera; triple band 
emission filter, triple band polychroic filter, three single band 
excitation filters; three widely spread dyes, namely blue, green and red), 
identity of the color spread matrix is precisely the assumption made. 

15 Similarly, in [SpeicherSS] , the dyes and epi - fluorescence cubes are chosen 
such that the color spread matrix is almost diagonal, so that, after 
thresholding, it can be approximated to the identity matrix. Therefore, 
the current standard techniques are nothing but simplifications of the 
color compensation approach. 

20 Also, in practice, one can choose the dyes and filters so 

that C is well conditioned; and obviously, the closer to diagonal one can 
make the color spread matrix, the less singular and ill-conditioned it 
will be. 

In the current implementation, using the set of 7 dyes, namely 
25 DAPI, FITC, CY3, CYS.S, CY5 , CyS . 5 , Cy7 , the exemplairy color spread and 
color compensation matrices are as follows: 



COLOR SPREAD MATRIX 



30 



0.740000 
0.080000 
0.060000 
0.010000 
0.050000 
0.060000 
0.000000 



0.000000 
0.840000 
0.150000 
0.010000 
0.000000 
0.000000 
0.000000 



0.000000 
0.070000 
0.600000 
0.330000 
0.000000 
0.000000 
0 .000000 



0. 

0. 

0 , 

0, 

0 

0 



010000 
060000 
230000 
650000 
060000 
000000 



0 .000000 



0.000000 
0 .020000 
0 .050000 
0.040000 
0 .680000 
0.210000 
0 .000000 



0.000000 
0.020000 
0 .020000 
0.040000 
0.050000 
0.860000 
0.010000 



0.000000 
0 .090000 
0 .100000 
0.120000 
0.160000 
0.140000 
0 .390000 



35 



1.059813 0.913197 0.785776 



SINGULAR VALUES 

0 .724402 0 .617623 



0.370161 0.340714 



COLOR COMPENSATION MATRIX (INVERSE MATRIX OF COLOR SPREAD MATRIX) 



1.350690 .-0.002264 0.014392 

-0.118809 1.209434 -0.101343 

40 -0.113836 -0.365231 2.091644 

0.048925 0.167552 -1.065018 

-0.098882 -0.014875 0,094547 



-0.025687 0.000258 0.000848 0,004326 

-0.072286 -0.018050 -0.018838 -0.216705 

-0.695213 -0.102289 0.000416 -0.196307 

1.900855 -0.019067 -0.062710 -0.320130 

-0.168748 1.497029 -0.074572 -0.556284 
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-0.070382 0.003806 -0.024192 0.043178 -0.367104 1.185891 -0.283059 
0.001805 -0.000098 0.000621 -0.001108 0.009413 -0.030407 2.571361 

Another practical consideration is that all computations (SVD 
calculations, storage of calibrated SVD matrix, SVD matrix multiplication) 
5 are preferably carried out at least on 16 bits fixed point integers, and 
more preferably on single precision floating point pixels. 

10 . Alternative Filter Conficrurationa 

In all of the above discussions, it has been assumed that 
interference filters (coated glass) were used. However, one could use 

10 liquid- crystal tunable filters (LCTF) instead of a multiband emission 
filter. The main advantage is electronically selectable center 
wavelength; the main disadvantages are very poor transmission efficiency 
(10-15%) and high cost ($15K) . Similarly, an alternative for the 
excitation filter wheel would be an acousto-optics tunable filter (AOTF) 

15 The main advantages are very fine bandwidth (1-Snm) , and an average 

transmission efficiency (40%) ; the main disadvantages are unavailability 
of off-the-shelf control electronics and cost. See (Morris94] for a 
review of both techniques . 

Yet another alternative to the excitation filter wheel, is the 

20 use of a tunable laser (dye laser) , which would provide very narrow 

bandwidth, electronically switched, at sufficiently high power. However, 
it is not clear that it could achieve the necessary spectral range (350- 
7S0nm) , and the cost would probaibly be prohibitive. 

11. Alternative Generation of Labeled Images 

25 The following discusses how the raw data is processed in order 

to generate the pseudo-colored, labeled metaphase image. 

Instead of solving the set of equations to perform color 
compensation, one could instead just try matching the spectral signatures 
to the actually observed values. Each dye, and each combination of dyes, 

30 for given filter sets, potentially generates a unique spectral signature 
(i.e., normalized set of intensities; for the pure dyes, these are the 
columns of the color spread matrix) . The label is then determined on the 
basis of which of those comes closest to the actual measured data. The 
first problem is that this requires calibrating not only all pure dyes, 

35 but also all combinations; this is a much more cumbersome process (at 

least 24x7=168 images to acquire) . Furthermore, combinations are bound to 
be much less stable, requiring in effect calibration for each experiment. 
Finally, even after calibration, this process is also computationally more 
expensive in order to obtain the processed dye images (for 7 dyes and 24 

40 combinations, there would be at least 336 multiplication and 168 additions 
per pixel) . 

Alternatively, again instead of solving the set of linear 
equations, one could perform cluster analysis on the acquired raw data. 
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The acquired raw pixels, when represented in N-dimensional space (N=number 
of dyes, counterstain, i.e., DAPI , excluded) should form Q clusters 
(Q=number of combinations, i.e., 24 or so). By identifying these Q 
clusters, and assigning all pixels to one of them, it should be possible 
5 to directly generate the map of all dye combinations. Rather than 

operating directly on the raw acquired images, they are preferably first 
normalized, to eliminate variations in absolute intensities (e.g., 
normalize with respect to the maximum value; this is similar to what needs 
to be done" when generating the map from the pure dye images) . 
Xo Cluster analysis is a fairly well covered topic, with 

available, published, algorithms. The major advantage of this approach, 
is that it does not require any calibration, and therefore is much less 
sensitive to experimental variations. 

On the other hand, it does not allow one to generate images of 
15 the pure dyes; more critically, one cannot know which cluster corresponds 
to which dye combination, that is to which chromosome. Another 
disadvantage is that it involves a rather huge data set (432K points in 
7 -dimensional space in the case of 768x576 images with 7 dyes) and hence 
would probably be computationally very expensive (this could be somewhat 
20 improved by eliminating all background pixels) . 

Another approach which does not require knowledge of the color 
spread matrix has been described in [Kawata96l . In very broad terms, one 
works with PxM equations (P=number of pixels, e.g., 430K, M=number of 
filter sets, e.g., 7) with PxM+N^ unJcnowns (N=number of dyes) . The 
25 unknowns are the pure dye data, plus the coefficients of the color spread 
matrix. To this one adds PxM+N^ inequalities expressing that the color 
spread matrix and the pure dye data are non- negative everywhere. Then, 
using eigenvalue analysis, a solution can be derived [Kawata96] . 
Advantages are that calibration is not needed, pure dye images can be 
30 obtained, and exact identification of dye combinations is possible. 
However, this obviously involves a rather huge data set, making it 
computationally cumbersome . This can be helped by eliminating all 
background pixels; further improvement can be made by considering only a 
small section of the image at a time. 
35 Yet another approach which does not require knowledge of the 

color spread matrix is an adaptation of the iterative blind deconvolution 
method (IBD) described in [AyersSS] . There, the authors were trying to 
estimate the original data from the knowledge of blurred observed data, 
without knowledge of the blurring function (point spread fxinction) . It is 
40 similar to the present problem where it is desired to know the original 
dye images from the "blurred" observed images without knowledge of the 
"blurring*^ function (color spread matrix) . This is achieved in a 
iterative fashion, by making use of the fact that both the dye data and 
the color spread matrix are non-negative everywhere. Briefly stated, 
45 starting from an initial arbitrary estimate of the raw dye data Fq (PxM 

matrix of the pure dye distributions for all pixels) , one can make a first 
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estimate Cq of the color spread matrix by computing Cq = I * Fq'-^ il = PxM 
matrix of the observed data for all pixels) . Cq is then modified to make 
all coefficients non-negative, and make all columns sum to 1. Then a new 
estimate Fj^ is computed by = Cq"-^ * X. F2 is then modified to make all 
5 values non-negative, conserving the total energy (i.e., spreading the sum 
of the absolute value of the negative elements of Fj_ over all positive 
elements) . The loop is then repeated with a new estimate etc. 
Advantages again are that calibration is not needed, pure dye images can 
be obtained, and exact identification of dye combinations is possible. 

10 However, again, the size of the matrices involved makes it computationally 
expensive. Here too, one can eliminate all backgrotind pixels and/or 
consider only a small section of the image at a time. In addition, there 
is no theoretical guarantee of uniqueness of the solution, or even of 
convergence of the algorithm (even though good results have been obtained 

15 in practice) . 

The above discussion is a reminder that the color compensation 
method does not absolutely require knowledge of the color spread matrix 
(and hence calibration) ; techniques exist which, while mathematically and 
computationally more complex, do not have this disadvantage. 

20 12. Non -Epi-illuiP '^ n Embodimants 

A limitation of almost all M-FISH solutions described above is 
the use of a polychroic mirror. This is actually a limitation not only 
for the instrument, but also for the wet preparation. Using such a filter 
limits the choice of dyes which can be used simultaneously; the excitation 
25 spectrum of one dye (which corresponds to a reflection band of the 

polychroic mirror) cannot overlap the emission spectrum of another (which 
should correspond to a transmission band of the polychroic mirror) . It 
forces one to use less than optimal excitation and emission filters,;, and 
therefore results in the very color spreading that requires efforts -to 
30 undo. Even when dichroic mirrors are used (as in [Speicher961 ) , the need 
to use up to 7 of them makes this technique very impractical. One could 
thus almost say that, from the instrument point of view, the principal 
culprit for the difficulties of implementation of M-FISH is the 
poly/dichroic mirror; eliminate it and the solution becomes much easier. 
35 Therefore a number of approaches which do not require an epi- fluorescence 
cube are described here. At present, however, the epi -illumination 
embodiment described in detail above remains preferred. 

FIG. 9 is a diagram of a trans - illumination fluorescence 
(brightf ield) microscope system 150. Elements corresponding to those in 
40 FIG. 1 are provided with corresponding reference numerals. Excitation 
radiation JErom source 25 passes through excitation filter 12 (one of 
several on an excitation filter wheel) , is reflected by a mirror 152 to a 
condenser 155. The excitation radiation passing through the condenser 
illuminates sample 10, and proceeds through microscope objective 17 to an 
45 emission filter 160 (one of several on an emission filter wheel) . The 
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excitation light traveling from the source, through the sample, and to 
(but not through) the emission filter is shown schematically by hollow 
. arrowheads . Fluorescent radiation emitted from sample 10 passes through 
objective 17 and through emission filter 20 to form an image in image 
5 plane 30. The fluorescent light traveling away from the sample is shown 
schematically by solid black arrowheads. As in the epi - illumination 
system, the image is digitized by CCD camera 3 2 and the digitized image is 
sent to computer 35 for subsequent processing. 

Thus, this embodiment still uses an excitation filter wheel 

10 containing single-band excitation filters as in the preferred epi- 

illumination embodiment. However, the multiple cubes are replaced by the 
emission filter wheel containing a plurality of N multiband emission 
filters (i.e,, no polychroic mirror is required) . Along the lines of the 
multi-cube embodiment, and according to the present invention, each 

15 emission filter is configured for an associated one of N subsets of the 
total set of dyes, with the i— siibset having a plurality of Mj_ distinct 
dyes (i ranges from 1 to N) . The sxibsets are chosen so that the j — 
siibset and the (j+l)-^^ subset have one dye in common, referred to as the 
common dye (j ranges from 1 to N-1) . 

20 This embodiment has the advantage that all that is required 

are single -band excitation filters (typically on a filter wheel) and a 
multiband emission filter wheel (no dichroic or polychroic mirror) . 

One disadvantage is that the illumination would not be as 
bright as with epi -illumination (the condenser, which has a lower 

25 nxunerical aperture, is used to focus the light onto the sample, instead of 
the objective itself) ; this effect gets worse for high power objectives, 
as they have higher numerical apertures (as a matter of fact, for low 
power, low NA objectives, trans -illumination is often brighter than epi- 
illumination) ; nevertheless, in the case of M-FISH, the sample should be 

30 reasonably bright (whole chromosome paints) and this should not be a major 

issue . 

Furthermore, such an arrangement cannot always fully reject 
the illumination light (there can easily be 10^ excitation photons for 
each emitted photon) ; it should be possible to deal with this using 

35 holographic notch filters (commonly used to address this problem in Raman 
spectroscopy, where the problem is much worse) . The main advantages are 
that one can use any dye (even if there is overlap between excitation 
spectrum of one and emission spectrum of another) , any number of them (10 
or 12 present no problem with filter wheels, and no particular problem 

40 computationally), filter pairs can be optimized independently, etc. It is 
even conceivable that one could use a single multiband emission filter 
(instead of a filter wheel), further simplifying the instrument. Epi- 
illumination is a technique which has worked in the past; recent 
improvements in the production of interference and holographic filters 

4 5 make it again potentially very useful (TaylorS?! . 
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FIG. 10 is a diagram of a darkfield fluorescence microscope 
system 170 that uses a darkfield condenser 172 (available for essentially- 
all microscopes) . The conventions of FIGS. 1 and 9 apply. The darkfield 
condenser produces an annular cone of illumination whose aperture is 
5 greater than the aperture of the objective. Therefore, unscattered 

excitation light does not enter the objective. Here too, the system uses 
excitation and emission filter wheels, and a darkfield condenser. As in 
the previously described trans - illumination (bright field) embodiment, each 
emission filter is configured for an associated one of N subsets of the 
10 total set of dyes, with the i— subset having a plurality of M^^ distinct 
dyes (i ranges from 1 to N) . The s\ibsets are chosen so that the j — 
subset and the (j+D— subset have one dye in common, referred to as the 
jth common dye (j ranges from 1 to N-1) , 

Compared to trans -illumination, this provides much better 
15 rejection of the excitation light, but at the cost of reduced brightness 
(less efficient condenser, reduced NA objective) . Advantages over epi- 
illumination are the same as for trans -illumination [Taylor87l . 

Yet another approach consists of delivering the excitation 
light via an optical fiber bundle external to the microscope, and oriented 
20 at a very shallow euigle with respect to the plsuae of the sample slide. 

The excitation light could be delivered either from above or below. the 
slide. Again, an emission filter wheel would be used where each emission 
filter is configured for an associated one of N subsets of the total set 
of dyes, with the i— siibset having a plurality of distinct dyes (i 
25 ranges from 1 to N) . The subsets are chosen so that the j— subset auid 
the (j+D— sxibset have one dye in common, referred to as the j— common 
dye (j ranges from 1 to N-1) . 

If delivery from underneath is practical, then the angle is 
preferably chosen below the critical angle (total reflection condition), 
30 to provide even further rejection of the excitation light (this is- the 

technique described as total internal reflection fluorescence microscopy 
by some authors [Axelrod87] ) . Such optical fiber deliveiry systems are 
widely available commercially, for a variety of light sources (xenon, 
mercury, etc); it would probably be required to design a lens to be fitted 
35 at the distal end of the fiber bundle, to focus the light on as small a 
spot as possible. Compared to trans- illumination, this provides much 
better rejection of the excitation light, while allowing comparable signal 
brightness. Advantages over epi- illumination are the same as for trans- 
illumination . 

40 X3 . References 

The following references are incorporated by reference in their entirety 
for all purposes as if fully set forth herein. 
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14 • Conclusion 

In conclusion, it can be seen that the present invention 
provides an elegant and cost-effective solution to the image registration 
problem in M-FISH and related applications. 

While the above is a complete description of specific 
embodiments of the invention, various modifications, alternative 
constructions, and equivalents may be used, a number of which were 
described above. Therefore, the above description should not be taken as 
limiting the scope of the invention as defined by the claims. 
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LJWiiT rq CLAIMED : 

2^ 1 , In an M-FISH fluorescence system having first and second 

2 optical entities for selective insertion into an optical path between a 

3 sample and an image plane, the first optical entity having a first 

4 multiband transmission characteristic configured to match the emission 

5 spectra of a first set of fluorescent dyes, the second optical entity 

6 having a second multiband transmission characteristic configured to match 

7 the emission spectra of a second set of fluorescent dyes, the improvement 

8 wherein: 

9 the first and second sets of dyes have a dye in common so as 

10 to facilitate the registration of images made with the first optical 

11 entity with images made with the second optical entity. 

1 2. The improvement of claim 1 wherein the system is an epi- 

2 fluorescence system. 

1 3 . The improvement of claim 1 wherein the first optical 

2 entity includes a multiband emission filter and a polychroic mirror. 

1 4. In an M-FISH epi -fluorescence system having first and 

2 second polychroic mirrors and first and second multiband emission filters, 

3 the first polychroic mirror and the first multiband emission filter being 

4 configured to match the excitation and emission spectra of a first set of 

5 fluorescent dyes, the second polychroic mirror and the second multibamd 
e emission filter being configured to match the excitation and emission 

7 spectra of a second set of fluorescent dyes, the improvement wherein: 

8 the first and second sets of dyes have a dye in common so as 

9 to facilitate the registration of images made with the first polychroic 

10 mirror and the first multiband emission filter with images made with the 

11 second polychroic mirror and the second multibamd emission filter. 

1 5 . A method of analyzing a sample to determine which portions 

2 of the sairple contain various ones of a set of fluorescent dyes, each dye 

3 being characterized by an excitation spectrtam and an emission spectrum, 

4 the method comprising the steps of : 

5 dividing the set of dyes into first and second svibsets having 

6 respective first and second pluralities of and M2 distinct dyes; 

7 the subsets of dyes being chosen so that the first and second 

8 subsets have one dye in common, referred to the common dye; 

9 providing a first polychroic mirror and a first multiband 

10 emission filter, the first polychroic mirror and the first multiband 

11 emission filter being configured to match the excitation and emission 

12 spectra of the first subset of dyes, and thus referred to as being 

13 associated with the first subset of dyes; 
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14 providing a second polychroic mirroi: and a second multiband 

15 emission filter, the second polychroic mirror and the second multiband 

16 emission filter being configured to match the excitation and emission 

17 spectra of the second subset of dyes, and thus referred to as being 

18 associated with the second subset of dyes; 

19 for each subset of dyes, generating a plurality of images by 

20 directing excitation radiation onto the sample along an optical path that 

21 includes that subset's associated polychroic mirror, and directing light 

22 emanating from the sample to an image plane along an optical path that 

23 includes that subset's associated polychroic mirror and associated 

24 emission filter, each of the images in said plurality of images being 

25 generated with excitation radiation in a respective one of a plurality of 

26 wavelength ranges corresponding to the excitation spectrum of a respective 

27 one of the distinct dyes in that subset , and thereby corresponding to a 

28 respective one of the distinct dyes in that subset; 

29 correlating (1) an image in the plurality of images generated 

30 with the first polychroic mirror that corresponds to the common dye with 

31 (2) an image in the plurality of images generated with the second 

32 polychroic mirror that corresponds to the common dye to generate a set of 

33 registration information; 

34 registering the pluralities of images, so generated, by using 

35 the set of registration information, so generated; and 

36 using the pluralities of images, so registered, to determine 

37 which portions of the sample contain various ones of the set of dyes. 

1 6 . The method of claim 5 , wherein : 

2 the first polychroic mirror has passbands and reflection 

3 bands ; 

4 the first emission filter has passbands corresponding to the 

5 passbands of the first polychroic mirror; and 

6 the plurality of wavelength ranges for the plurality of images 

7 corresponding to the first subset of dyes lie in the reflection bands of 

8 the first polychroic mirror. 

1 7. The method of claim 5, wherein said step of using 

2 comprises, for at least one image that corresponds to a particular dye in 

3 a particular siibset, correcting measured pixel values in the image for the 

4 effect of emissions from other dyes in the particular subset. 

1 8. The method of claim 7, wherein said step of correcting 

2 measured pixel values comprises using calibration information developed by 

3 generating, and analyzing a plurality of images, each image of which is 

4 generated with a respective sample having portions containing a respective 

5 single dye and sxibstantially no portions containing any other dye. 
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9. A method of analyzing a sample to determine which portions 

2 of the sample contain various ones of a set of fluorescent dyes, each dye 

3 being characterized by an excitation spectrum and an emission spectrum, 

4 the method comprising the steps of: 

5 dividing the set of dyes into a plurality of N subsets, a 

6 given one of the subsets of dyes being denoted the i^ subset, where i 

7 ranges from 1 to N, with the i^ subset having a plurality of Mj^ distinct 

8 dyes; 

9 the subsets of dyes being chosen so that the i— subset and 

10 the (i + D— subset have one dye in common, referred to the i— common dye? 

12, providing a plurality of N polychroic mirrors, each associated 

12 with a respective subset of dyes; 

3_3 the i— polychroic mirror being configured to match the 

14 excitation spectra and emission spectra of the i— siibset of dyes ; 

15 providing a plurality of N multiband emission filters, each 

16 associated with a respective polychroic mirror; 

17 the i— multiband emission filter being configured to match 

18 the emission spectra of the subset of dyes; 

19 providing a set of single-band excitation filters configured 

20 to match the reflection bands of the plurality of polychroic mirrors; 

21 for values of i from 1 to N, 

22 generating a plurality of images by directing 

23 excitation radiation onto the sample along an optical path that 

24 includes a selected one of excitation filters and the i— 

25 polychroic mirror, and directing light emanating from the sample to 

26 an image plane along an optical path that includes the i— 

27 polychroic mirror and the i— emission filter, each of the images in 
2 8 said plurality of M^^ images thereby corresponding to a respective 

29 one of the distinct dyes; 

30 for values of i from 1 to (N-1) , 

31 correlating (1) an image in the i— pliirality of images 

32 that corresponds to the i^ common dye with (2) an image in the 

33 (i+1)^^ plurality of images that corresponds to the i— common dye 

34 to generate a set of registration information, referred to as the 

35 ±^ registration information; 

36 registering the pluralities of images, so generated, by using 

37 the sets of registration information, so generated; and 

38 using the pluralities of images, so registered, to determine 

39 which portions of the sample contain various ones of the set of 

40 fluorescent dyes. 

1 .10. The method of claim 9, wherein said step of registering 

2 the pluralities of images comprises; for values of i from 1 to (N-1) , 

3 registering the images in the (i+l)^ plurality of images with the images 

4 in the plurality of images using the i— registration information. 
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1 11. The mechod of claim 9, wherein said step of using 

2 comprises, for at least one image that corresponds to a particular dye in 

3 a particular subset, correcting measured pixel values in the image for 

4 the effect of emissions from other dyes in the particular sxibset . 

1 12. The method of claim 11, wherein said step of correcting 

2 measured pixel values comprises using calibration information developed by 

3 generating and analyzing a plurality of images, each image of which is 

4 generated with a respective sample having portions containing a respective 

5 single dye and substantially no portions containing any other dye. 

1 13. Apparatus for use with a fluorescence microscope for 

2 analyzing a sample to determine which portions of the sample contain 

3 various ones of a set of fluorescent dyes, each dye being characterized by 

4 an excitation spectrum and an emission spectrum, the apparatus comprising: 

5 a first polychroic mirror and a first multiband emission 

6 filter, the first polychroic mirror and the first multiband emission 

7 filter being configured to match the excitation and emission spectra of a 

8 first subset of dyes, and thus referred to as being associated with the 

9 first subset of dyes; 

10 a second polychroic mirror and a second multiband emission 

11 filter, the second polychroic mirror and the second multiband emission 

12 filter being configured to match the excitation and emission spectra of a 

13 second subset of dyes, and thus referred to as being associated with the 

14 second subset of dyes; 

15 each subset having a plurality of distinct dyes, the subsets 

16 of dyes being chosen so that the subsets have one dye in common, referred 

17 to the common dye; 

18 an illumination subsystem for selectively directing excitation 

19 radiation in respective ones of a plurality of wavelength ranges 

20 corresponding to the excitation spectra of respective ones of the distinct 

21 dyes in the set of dyes toward along a first portion of an optical path in 

22 the fluorescence microscope; and 

23 a mirror support structure for selectively maintaining a 

24 desired polychroic mirror and multiband emission filter in a second 

25 portion of the optical path. 

1 14 . The apparatus of claim 13 wherein the illumination 

2 subsystem comprises: 

3 a broadbamd light source; 

4 a set of single -band excitation filters configured to mat^ch 

5 the absorption bands of the set of dyes ; and 

6 an excitation filter support structure for selectively 

7 maintaining a desired excitation filter in a first portion of an optical 

8 path in the fluorescent microscope 
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1 15. The apparatus of claim 14 wherein the excitation filter 

2 support structure comprises a detented rotatable wheel with apertures 

3 accommodating the excitation filters. 

1 16 . Apparatus for use with a fluorescence microscope for 

2 analyzing a sample to determine which portions of the sample contain 

3 various ones of a set of fluorescent dyes, each dye being characterized by 

4 an excitation spectrum and an emission spectrum, the apparatus comprising: 

5 a plurality of N polychroic mirrors, each associated with a 

6 respective one of N subsets of dyes, a given one of the subsets of dyes 

7 being denoted the i— subset, where i ranges from 1 to N, with the i^ 

8 siibset having a plurality of distinct dyes; 

9 the subsets of dyes being chosen so that the i^ subset and 

10 the (i+1)^^ subset have one dye in common, referred to the i— common dye; 

11 the i— polychroic mirror being configured to match the 

12 excitation spectra and emission spectra of the i— subset of dyes; 

13 a plurality of N multiband emission filters, each associated 

14 with a respective polychroic mirror; 

15 the i— multiband emission filter being configured to match 

16 the emission spectra of the i^ subset of dyes; 

17 a set of single -band excitation filters configured to match 

18 the reflection bands of the plurality of polychroic mirrors; 

10 an excitation filter support structxire for selectively 

20 maintaining a desired excitation filter in a first portion of an optical 

21 path in the fluorescent microscope; and 

22 a mirror support structure for selectively maintaining a 

23 desired polychroic mirror and the associated multiband emission filter in 

24 a second portion of the optical path in the fluorescence microscope. 

1 17 , The apparatus of claim 16 wherein the excitation filter 

2 support structure comprises a detented rotatable wheel with apertures 

3 accommodating the excitation filters. 

1 18 . The apparatus of claim 16 wherein the mirror support 

2 structure comprises a detented rotatable turret . 

1 19 . The apparatus of claim 16 wherein the mirror support 

2 structure comprises a detented movable slide. 

1 20. A method of generating a labeled image from a plurality 

2 of images, referred to as dye images and associated with respective dyes, 

3 each dye image having an array of pixel values with at least one region of 

4 non-zero pixel values indicating the presence of that dye image's 

5 associated dye in that region, the method comprising: 

6 normalizing at least some pixel values within each dye image 

7 to provide a plurality of associated normalized images, each normalized 
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8 image having an array of normalized pixel values spatially corresponding 

9 to the associated dye image's array of pixel values; 

10 ratioing at least some pixel values within each normalized 

11 image by dividing the spatially corresponding normalized pixel value in 

12 that normalized image by the sum of the spatially corresponding normalized 

13 pixel values in the plurality of normalized images; and 

14 labeling a pixel in the labeled image with a given dye if and 

15 only if the pixel value for the spatially corresponding pixel in the given 

16 dye's corresponding ratio image is above a threshold. 

1 21- The method of claim 20 wherein said step of normalizing 

2 includes, for at least one of the dye images, creating a mask defining 

3 regions that are likely to have non-zero pixel values. 

1 22. The method of claim 21 wherein said step of creating a 

2 mask includes a single mask for all the dye images . 

1 23 . The method of claim 22 wherein: 

2 a particular one of the dyes stains all objects of interest; 

3 the single mask is derived from a dye image associated with 

4 the particular one of the dyes. 

1 24 . The method of claim 22 wherein the mask is derived from 

2 the plurality of dye images, and has ON pixels at positions where any of 

3 the dye images has a non-zero pixel value. 

1 25. The method of claim 20 wherein said step of labeling a 

2 pixel includes establishing a set of different colors, each uniqfuely 

3 corresponding to a different one or combination of the dyes . 

1 26. The method of claim 20 wherein said steps of normalizing 

2 and ratioing are carried out for only pixels falling within a mask. 

1 27 . A method for generating a labeled image based on a 

2 plurality of dye images, the method comprising: 

3 normalizing pixel intensities within each dye image; 

4 ratioing normalized pixel intensities by dividing each 

5 normalized pixel value by the sum, taken over the normalized pixel values 

6 for all the different dyes, of the pixel values for that pixel position; 

7 and 

8 labeling pixels in the labeled image according to which 

9 ratioed normalized pixel values are above a threshold. 

1 28, The method of claim 27, and further comprising colorizing 

2 the labeled pixels according to the combinations of dyes considered to 

3 have labeled that pixel . 
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